The Logic of Exploratory and Confirmatory Data Analysis

Exploratory Data Analysis (EDA) and Confirmatory Data Analysis (CDA) are two statistical methods widely used in scientific research. They are typically applied in sequence: first, EDA helps form a model or a hypothesis to be tested, and then CDA provides the tools to confirm if that model or hypothesis holds true. When both analyses are applied within a single experiment, two main types of errors can occur that fall under the general term of selection bias. One error is the biased selection of the set of data used to confirm the model derived by the EDA. The other error occurs when CDA becomes part of EDA instead of being applied after EDA completion. As a result of selection bias, overfitting of a model can occur in a manner that makes the model stand true only narrowly, i.e. for the specific sample from which it was derived, without any generalizability. This bias in planning the analysis occurs frequently in the literature. This paper provides the theoretical background and the conceptual tools by which to identify such errors in the literature and to carry out the analysis properly. Applications of EDA and 35 exploratory and Confirmatory data analysis CDA in medical biomarker research are used as paradigms for clarification of concepts. “έστιν ουν επιστήμη δόξα αληθής μετά λόγου” (Science is affirmed knowledge through logical arguments.)